Transparent reporting of multivariable prediction models for individual prognosis or diagnosis: checklist for systematic reviews and meta-analyses (TRIPOD-SRMA)

Most clinical specialties have a plethora of studies that develop or validate one or more prediction models, for example, to inform diagnosis or prognosis. Having many prediction model studies in a particular clinical field motivates the need for systematic reviews and meta-analyses, to evaluate and summarise the overall evidence available from prediction model studies, in particular about the predictive performance of existing models. Such reviews are fast emerging, and should be reported completely, transparently, and accurately. To help ensure this type of reporting, this article describes a new reporting guideline for systematic reviews and meta-analyses of prediction model research.


Methods
The following items relate to the Methods.

Item 5: Eligibility criteria
Specify study characteristics (e.g. in relation to PICOTS), report characteristics (e.g. years considered, language, publication status) and prediction model specific aspects (e.g. specific predictor(s), outcome(s), whether development and validation studies are eligible or only validation studies), used as criteria for eligibility.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: • Definitely Agree • Mostly Agree • Neither Agree nor Disagree • Mostly Disagree • Definitely Disagree Make a comment on your choice here:

Specify all databases, registers, websites, organisations, reference lists and other sources searched or consulted to identify studies. Specify the date when each source was last searched or consulted.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: • Definitely Agree • Mostly Agree • Neither Agree nor Disagree • Mostly Disagree • Definitely Disagree Make a comment on your choice here:

Item 8: Selection process
Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: • Definitely Agree • Mostly Agree • Neither Agree nor Disagree • Mostly Disagree • Definitely Disagree Item 9: Data collection process Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: • Definitely Agree • Mostly Agree • Neither Agree nor Disagree • Mostly Disagree • Definitely Disagree Make a comment on your choice here:

Item 10: Data items
List and define all variables for which data were sought from each study (e.g. PICOTS, study dates, follow-up duration, country, funding

sources, conflicts of interest).
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following:

Item 17: Synthesis of results
If meta-analysis was carried out, describe the methods for pooling performance measures for each model, including how any heterogeneity in model performance was handled (e.g. using random effects) and quantified (e.g. tau-squared, prediction intervals).
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Results
The following items relate to the Results.

Item 20: Study selection
Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram. Also provide the number of models and the number and type of validation in the included studies.
To what extent do you agree that this item should be included in the Choose one of the following answers Please choose only one of the following: To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here: Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Report the results from any applicability assessment (e.g. PROBAST) separately for each included model and all validations in each study.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Present for each model development study: model equation (or link to source code, website, app etc.), estimates (and confidence intervals) of the model's performance measures from apparent or internal validation. Present for each model validation study, estimates of performance measures (and confidence intervals). Note any changes from the original development study.
To what extent do you agree that this item should be included in the Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Present results of each meta-analysis (if done), including summary estimates, confidence intervals and measures of heterogeneity in the model's performance measures. Forest plots may be useful.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Report results from any subgroup, sensitivity or meta-regression analyses (see Item 19).
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Discussion
The following items relate to the Discussion.

Summarise the main findings (for each model if relevant) including the strength of evidence (based on external validation performance) and heterogeneity (e.g. in performance across studies).
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Item 28: Limitations
Discuss the strengths and limitations at study and model level (e.g. risk of bias) and at review level (e.g. incomplete retrieval of identified research, reporting bias).
To what extent do you agree that this item should be included in the Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Item 29: Implications
Discuss the findings in the context of other evidence and in the context of the objectives. Consider the relevance to key groups (e.g. healthcare providers, users and policy makers). Discuss any implications/suggestions for potential clinical use and future research.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Provide a brief summary of key findings, implications and future research.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Other Information
The following items relate to Other Information.

Item 31: Registration
Provide registration information for the review, including register name and registration number, or state that the review was not registered.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here:

Item 32: Protocol
Indicate where the review protocol can be accessed, or state that a protocol was not prepared.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following:

Item 33: Support and COI
Describe sources of financial or non-financial support for the review (e.g. supply of data), and the role of the funders or sponsors in the review. Declare any potential conflicts of interest for the review authors.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: Make a comment on your choice here: Item 34: Availability of data, code, and other materials Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.
To what extent do you agree that this item should be included in the checklist?
Choose one of the following answers Please choose only one of the following: • Definitely Agree • Mostly Agree • Neither Agree nor Disagree • Mostly Disagree • Definitely Disagree

Comments
Are there any items not currently included in TRIPOD-SRMA that you think should be included?
Please write your answer here: Do you have any other comments or suggestions for TRIPOD-SRMA?
Please write your answer here: Acknowledgement Finally, do you consent to be acknowledged by name for your contribution in any resulting academic publications? * Choose one of the following answers Please choose only one of the following: We appreciate your participation in the Delphi survey for TRIPOD-SRMA. Responses to this survey will be discussed within the TRIPOD group and will inform which items are included in the checklist for the next Delphi survey. All comments will be carefully considered. Thank you for your time.
Reporting guidelines for systematic reviews and metaanalyses of prediction model studies (TRIPOD-SRMA)

Delphi survey results -Round 1
Thank you for participating in Round 1 of our Delphi study to help develop a reporting guideline for systematic reviews and meta-analyses of prediction model studies. Your responses and comments have been very helpful to us as we revise the draft checklist and prepare the accompanying guidance documents.
We are pleased to say that the vast majority of Delphi responders agreed with the inclusion of all items in the draft checklist. However, we have now made modifications to both the items included/excluded and the wording of items, based on Delphi participant feedback.
We are pleased to share a summary of the results from the first Delphi survey.
Out of 86 individuals invited to participate in the Delphi survey, 43 (50%) responded. The experience of responders is given in Table 1 below:

Experience in
Number of participants who said "yes" Developing and validating prognostic prediction models using primary studies Other 6 (other types of systematic reviews including overall prognosis or prognostic factor studies, methodology, IPD meta-analysis) Below is a summary of the Delphi responses to the questions asking participants to state the level of agreement to each item in the checklist (DA=Definitely agree, MA=Mostly Agree, N=Neither agree nor disagree, MD=Mostly disagree, DD=Definitely disagree).
TRIPOD-SRMA Checklist items DA MA N MD DD Item 1: Title Identify the report as a systematic review or meta-analysis (or both) of diagnostic or prognostic model studies. Specify the target population and outcome(s) predicted.

36
7 0 0 0 Item 8: Selection process Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process. 34 7 2 0 0 Item 9: Data collection process Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and if applicable, details of automation tools used in the process. Item 28: Limitations Discuss the strengths and limitations at study and model level (e.g. risk of bias) and at review level (e.g. incomplete retrieval of identified research, reporting bias). Item 31: Registration Provide registration information for the review, including register name and registration number, or state that the review was not registered. 34 7 1 1 0 Item 32: Protocol Indicate where the review protocol can be accessed, or state that a protocol was not prepared. 32 8 3 0 0 Item 33: Support and COI Describe sources of financial or non-financial support for the review (e.g. supply of data), and the role of the funders or sponsors in the review. Declare any potential conflicts of interest for the review authors. 40 3 0 0 0 Item 34: Availability of data, code, and other materials Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review.
28 9 5 1 0 In line with previous Delphi studies conducted by the group, we considered consensus to be achieved if two thirds of participants agreed with the item's inclusion (if participants 'definitely agreed' or 'mostly agreed') or if two thirds disagreed with the item's inclusion (if participants 'definitely disagreed' or 'mostly disagreed'). By this definition, we achieved consensus on the inclusion of every item in the checklist with a minimum agreement of 76% for any individual item.
Although consensus was achieved, we have modified the checklist items and wording based on the comments left by the participants, as we felt these feedback comments were particularly helpful in the development of the checklist.
For instance, we made the following changes following the feedback: • Abstract: We have now included a separate checklist in line with PRISMA 2020. Feedback was that the item was not specific enough and separate guidance for writing abstracts would be preferable.
• Objectives: We have removed mention of PICOTS from this item as several responses suggested that it wouldn't always be the right structure for all possible reviews of prediction models. We have also amended wording to be clearer that a comparator model is not always relevant.
• • Data transformation: General feedback on this item was that it should be removed or merged with another item. Therefore, we have removed this item and included it with the item for synthesis methods (see new item 14 below).
• Dealing with unreported information: Feedback on the item was that it contained too much detail and that there was overlap with the earlier item on the data collection process. Therefore, we have simplified the item wording and made it more distinct from the earlier item.
Item 11: Dealing with unreported information Describe how any required but unreported information was handled (e.g. contacted authors, calculated from other reported information).
• Risk of bias and applicability: Feedback suggested we combine the items on risk of bias assessment and applicability and not mention any specific risk of bias tools. Therefore, the two previous items have now been merged into one and PROBAST is no longer mentioned as an example of a risk of bias tool. • Additional analyses: Feedback suggested that authors should have to distinguish between planned and unplanned analyses. Therefore, we have amended the wording.
Item 15: Additional analyses Describe any subgroup, sensitivity or meta-regression analyses, including whether preplanned.
• Certainty assessment: There were suggestions to not name specific tools, hence we have removed the example suggesting an adaptation of GRADE.
• Results item on study characteristics: We now include both study and model characteristics to match the item in the methods. We have also simplified the text as suggested in the feedback, by removing the examples which will be discussed in the E&E. • Additional items: We have now added two new items in the results relating to heterogeneity investigation and certainty of evidence. Although items were included in the methods for these, feedback from participants highlighted that the corresponding items in the results were missing.
• Discussion: Based on the feedback we received, we have modified the items in the discussion to summarise the findings, strengths and limitations of the evidence, and then have a separate item relating to the limitations of the review. This is also in line with PRISMA 2020.
• Conclusion: We have removed the item for conclusions at the end of the discussion, as feedback suggested this was not required in a reporting checklist.
Again, we thank all the Delphi participants for their vital feedback, and we invite you to take part in the second and final Delphi round for TRIPOD-SRMA, in which you will have further opportunity to comment on the revised checklist items.

Development of a reporting guideline for systematic reviews and meta-analyses of prediction model studies (TRIPOD-SRMA)
Second Delphi round

Instructions:
Checklist items are listed below and you are asked to provide any additional feedback you have for each item. Please note that in this Delphi round, we are not asking you to state whether you think the items should/should not be included.

Title and Abstract
The following items relate to the Title and Abstract.

Identify the report as a systematic review or meta-analysis (or both) of diagnostic or prognostic model studies. Specify the target population and outcome(s) predicted as relevant to the review question.
(modified item from another checklist e.g. PRISMA or TRIPOD-Cluster)

Do you have any comments about this item?
Please write your answer here:

Item 2: Abstract
See the TRIPOD-SRMA Checklist for Abstracts.

Included studies 7
Give the total number of included studies and models, and summarise relevant study characteristics and model details.

8
Present results for the main models of interest. If meta-analysis was used to synthesise study estimates of model performance, report the summary result and confidence/credible interval for each performance measure.

Limitations of evidence 9
Provide a brief summary of the limitations of the evidence included in the review.
Interpretation 10 Provide a general interpretation of the results and important implications for research and practice.

Funding 11
Specify the primary source of funding for the review.